1. Field of the Invention
This invention relates to video systems and composition of multiple digital images. The invention is more particularly related to the composition of multiple digital images each captured by an individual camera of a camera array, and to a method of warping at least portions of images so that the images can be combined, without overlap or inconsistent pixels, into a single image. The invention also related to the registration of multiple cameras and the determination of a transformative equation that allows fast warping and summation of plural images from the multiple cameras to produce a single image. The invention is also particularly related to digital panning and zooming of a scene captured by a multiple camera array, and the automatic control of pan and zoom of an individually controlled camera or camera array.
2. Discussion of the Background
Remote and locally located cameras typically include devices for camera control. Devices include stepping motors or other mechanisms configured to point the camera or an image capturing device toward a scene or point of interest. Examples include teleconferencing applications, surveillance cameras, security cameras, cameras that are remotely controlled, activated by motion, light or other stimuli, remote sensing cameras such as those placed on robotic means (examples including those used in space exploration, deep sea diving, and for sensing areas or scenes to dangerous or inaccessible for normal camera operations (inside nuclear reactor cores, inside pipes, police cars, or law enforcement robotics, for example).
Normally, cameras are manually operated by a human operator on site, or remotely controlling the camera via a steering input (joystick or mouse, for example). In the case of remotely steered cameras, steering inputs generally activate a control program that sends commands to a stepping motor or other control device to steer a camera toward an object, item, or area of interest. General zooming functions of the camera may also be activated either on site or remotely.
In the case of teleconferencing applications (meetings, lectures, etc.), a variable angle camera with a mechanical tilt pan, focal length, and zoom capability is normally used. Such devices generally require a human operator to orient, zoom, and focus a video or motion picture camera. In some cases, conference participants may be required to activate a specific camera or signal attention of a camera configured to zoom in or focus on selected areas of a conference room.
Multiple cameras have been utilized in a number of applications. For example, Braun et al., U.S. Pat. No. 5,187,571, “TELEVISION SYSTEM FOR DISPLAYING MULTIPLE VIEWS OF A REMOTE LOCATION,” teaches an NTSC camera array arranged to form an aggregate field, and Henley, U.S. Pat. No. 5,657,073, “SEAMLESS MULTI-CAMERA PANORAMIC IMAGING WITH DISTORTION CORRECTION AND A SELECTABLE FIELD OF VIEW,” teaches a system for production of panoramic/panospheric output images.
Applications for multiple or steerable cameras include teleconferencing systems that typically direct a camera toward a speaker who is then broadcast to other teleconference participants. Direction of the camera(s) can be performed manually, or may utilize a tracking mechanism to determine a steering direction. Some known tracking mechanisms include, Wang et al., “A Hybrid Real-Time Face Tracking System,” in Proc. ICASSP 98, and, Chu, “Superdirective Microphone Array for a Set-Top Videoconferencing System,” In Proc. ICASSP 97.
However, technical challenges and costs have prevented such systems from becoming common and in wide spread use.
Systems attempting to integrate multiple images have failed to meet the needs or goals of users. For) example, McCutchen, U.S. Pat. No. 5,703,604, “IMMERSIVE DODECAHEDRAL VIDEO VIEWING SYSTEM,” teaches an array of video cameras arrayed in a dodecahedron for a complete spherical field of view. Images are composed at the receiving end by using multiple projectors on a hemispherical or spherical dome. However, the approach taught in McCutchen will suffer problems at image boundaries, as the multiple images will not register perfectly and result in obvious “seams.”
In another example, Henley et al., U.S. Pat. No. 5,657,073, “SEAMLESS MULTI-CAMERA PANORAMIC IMAGING WITH DISTORTION CORRECTION AND SELECTABLE FIELD OF VIEW,” teaches combination of images from radially-arranged cameras. However, Henley fails to disclose any but radially-arranged cameras, and does not provide details on image composition methods.